{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 0
   },
   "source": [
    "# Data Preprocessing\n",
    ":label:`sec_tablesaw`\n",
    "\n",
    "So far we have introduced a variety of techniques for manipulating data that are already stored in `NDArray`s.\n",
    "To apply deep learning to solving real-world problems,\n",
    "we often begin with preprocessing raw data, rather than those nicely prepared data in the `NDArray` format.\n",
    "Among popular data analytic tools in Java, the `tablesaw` package is commonly used. If you have used the pandas package for Python, you will find this familiar.\n",
    "So, we will briefly walk through steps for preprocessing raw data with `tablesaw`\n",
    "and converting them into the `NDArray` format.\n",
    "We will cover more data preprocessing techniques in later chapters.\n",
    "\n",
    "\n",
    "## Adding tablesaw dependencies to Jupyter notebook\n",
    "\n",
    "You can add tablesaw dependencies by adding a Java cell including:\n",
    "\n",
    "```\n",
    "%%loadFromPOM\n",
    "<dependency>\n",
    "    <groupId>tech.tablesaw</groupId>\n",
    "    <artifactId>tablesaw-jsplot</artifactId>\n",
    "    <version>0.38.1</version>\n",
    "</dependency>\n",
    "```\n",
    "\n",
    "To make it easy to include tablesaw in jupyter notebook, we create an utility notebook that can be loaded by:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load ../utils/plot-utils.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading the Dataset\n",
    "\n",
    "As an example, we begin by creating an artificial dataset that is stored in a\n",
    "csv (comma-separated values) file `../data/house_tiny.csv`. Data stored in other\n",
    "formats may be processed in similar ways.\n",
    "\n",
    "Below we write the dataset row by row into a csv file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load ../utils/djl-imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "File file = new File(\"../data/\");\n",
    "file.mkdir();\n",
    "\n",
    "String dataFile = \"../data/house_tiny.csv\";\n",
    "\n",
    "// Create file\n",
    "File f = new File(dataFile);\n",
    "f.createNewFile();\n",
    "\n",
    "// Write to file\n",
    "try (FileWriter writer = new FileWriter(dataFile)) {\n",
    "    writer.write(\"NumRooms,Alley,Price\\n\"); // Column names\n",
    "    writer.write(\"NA,Pave,127500\\n\");  // Each row represents a data example\n",
    "    writer.write(\"2,NA,106000\\n\");\n",
    "    writer.write(\"4,NA,178100\\n\");\n",
    "    writer.write(\"NA,NA,140000\\n\");\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 2
   },
   "source": [
    "To load the raw dataset from the created csv file,\n",
    "we import the `tablesaw` package and invoke the `read` function to read directly from the csv we created.\n",
    "This dataset has four rows and three columns, where each row describes the number of rooms (\"NumRooms\"), the alley type (\"Alley\"), and the price (\"Price\") of a house.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Table data = Table.read().file(\"../data/house_tiny.csv\");\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 4
   },
   "source": [
    "## Handling Missing Data\n",
    "\n",
    "Note that there are some blank spaces which are missing values.\n",
    "To handle missing data, typical methods include *imputation* and *deletion*,\n",
    "where imputation replaces missing values with substituted ones,\n",
    "while deletion ignores missing values. Here we will consider imputation.\n",
    "\n",
    "We split the `data` into `inputs` and `outputs` by creating new tables and specifying the columns desired, where the former takes the first two columns while the latter only keeps the last column.\n",
    "For numerical values in `inputs` that are missing, we replace the missing data entries with the mean value of the same column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "Table inputs = data.create(data.columns());\n",
    "inputs.removeColumns(\"Price\");\n",
    "Table outputs = data.select(\"Price\");\n",
    "\n",
    "Column col = inputs.column(\"NumRooms\");\n",
    "col.set(col.isMissing(), (int) inputs.nCol(\"NumRooms\").mean());\n",
    "inputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 6
   },
   "source": [
    "For categorical or discrete values in `inputs`, we consider missing data or null as a category.\n",
    "Since the \"Alley\" column only takes two types of categorical values \"Pave\" and an empty string which represents missing data/null,\n",
    "`tablesaw` can automatically convert this column to two columns.\n",
    "We will modify these two columns to assign a name to them which will be \"Alley_Pave\" and \"Alley_nan\".\n",
    "A row whose alley type is \"Pave\" will set values of \"Alley_Pave\" and \"Alley_nan\" to true and false.\n",
    "A row with a missing alley type will set their values to false and true.\n",
    "After this, we will add these columns to the original data/table but converting them to double so it changes true and false to 1 and 0 respectively. Finally, we remove the original column \"Alley\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "StringColumn col = (StringColumn) inputs.column(\"Alley\");\n",
    "List<BooleanColumn> dummies = col.getDummies();\n",
    "inputs.removeColumns(col);\n",
    "inputs.addColumns(DoubleColumn.create(\"Alley_Pave\", dummies.get(0).asDoubleArray()), \n",
    "                  DoubleColumn.create(\"Alley_nan\", dummies.get(1).asDoubleArray())\n",
    "                 );\n",
    "inputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 8
   },
   "source": [
    "## Conversion to the NDArray Format\n",
    "\n",
    "Now that all the entries in `inputs` and `outputs` are numerical, they can be converted to the `NDArray` format.\n",
    "Once data are in this format, they can be further manipulated with those NDArray functionalities that we have introduced in :numref:`sec_NDArray`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "NDManager nd = NDManager.newBaseManager();\n",
    "NDArray x = nd.create(inputs.as().doubleMatrix());\n",
    "NDArray y = nd.create(outputs.as().intMatrix());\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 12
   },
   "source": [
    "## Summary\n",
    "\n",
    "* Like many other extension packages in the vast ecosystem of Java, `tablesaw` can work together with `NDArray`.\n",
    "* Imputation and deletion can be used to handle missing data.\n",
    "\n",
    "\n",
    "## Exercises\n",
    "\n",
    "Create a raw dataset with more rows and columns.\n",
    "\n",
    "1. Delete the column with the most missing values.\n",
    "2. Convert the preprocessed dataset to the `NDArray` format.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Java",
   "language": "java",
   "name": "java"
  },
  "language_info": {
   "codemirror_mode": "java",
   "file_extension": ".jshell",
   "mimetype": "text/x-java-source",
   "name": "Java",
   "pygments_lexer": "java",
   "version": "14.0.2+12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
